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Abstract 

Approach and landing operations during periods 
of reduced visibility have plagued aircraft pilots since 
the beginning of aviation. Although techniques are 
currently available to mitigate some of the visibility 
conditions, these operations are still ultimately 
limited by the pilot’s ability to “see” required visual 
landing references (e.g., markings and/or lights of 
threshold and touchdown zone) and require 
significant and costly ground infrastructure. Certified 
Enhanced Flight Vision Systems (EFVS) have shown 
promise to lift the obscuration veil. They allow the 
pilot to operate with enhanced vision, in lieu of 
natural vision, in the visual segment to enable 
equivalent visual operations (EVO). 

An aviation standards document was developed 
with industry and government consensus for using an 
EFVS for approach, landing, and rollout to a safe taxi 
speed in visibilities as low as 300 feet runway visual 
range (RVR). These new standards establish 
performance, integrity, availability, and safety 
requirements to operate in this regime without 
reliance on a pilot’s or flight crew’s natural vision by 
use of a fail-operational EFVS. A pilot-in-the-loop 
high-fidelity motion simulation study was conducted 
at NASA Langley Research Center to evaluate the 
operational feasibility, pilot workload, and pilot 
acceptability of conducting straight-in instrument 
approaches with published vertical guidance to 
landing, touchdown, and rollout to a safe taxi speed 
in visibility as low as 300 feet RVR by use of vision 
system technologies on a head-up display (HUD) 
without need or reliance on natural vision. Twelve 
crews flew various landing and departure scenarios in 
1800, 1000, 700, and 300 RVR. This paper details the 
non-normal results of the study including objective 
and subjective measures of performance and 
acceptability. The study validated the operational 
feasibility of approach and departure operations and 
success was independent of visibility conditions. 


Failures were handled within the lateral confines of 
the runway for all conditions tested. The fail- 
operational concept with pilot in the loop needs 
further study. 

Introduction 

Ever since the Wright Brothers were tired of 
flying in circles around Huffman Field and wanted to 
use the airplane to go somewhere, the problem of 
reduced visibility has been an issue. Lindbergh said 
“Aviation will never amount to much until we learn 
to free ourselves from the mists. What I really need is 
a pair of spectacles to see through the fog. If I had a 
device like that, how simple the entire flight would 
be!” 1 Even though aviation has likely eclipsed his 
imagination, the problem of reduced visibility still 
plagues the industry and complex instrument landing 
systems are used as crutches to replace pilot vision 
for landing. Currently, the U.S. air transportation 
system is undergoing a transformation to 
accommodate the movement of large numbers of 
people and goods in a safe, efficient, and reliable 
manner. 2 One of the key capabilities envisioned to 
achieve this Next Generation Air Transportation 
System (NextGen) is the concept of equivalent vision 
operations (EVO). EVO is the ability to achieve the 
safety of current-day Visual Flight Rules (VFR) 
operations and maintain the operational tempos of 
VFR irrespective of the weather and visibility 
conditions. As visibilities decrease, instrument 
landing systems become increasingly complex and 
require: 

• additional training, experience, and 
currency for the crew 

• additional aircraft systems including auto 
land and auto throttle with associated 
maintenance costs 



• additional airport ground infrastructure 
including paint and signage and 
sophisticated approach and centerline 
lighting systems often with individual 
controls 

• additional procedure design and charting 
procedures often with multiple procedures 
for each runway end 

Each of these items represents a significant 
burden to operators, airports, and regulators; and 
collectively are one reason for the limited availability 
of low visibility instrument approach procedures. As 
of April 2015, there were 1,277 Instrument Landing 
System (ILS) instrument approach procedures (IAPs) 
to Category I minima (no lower than 200 ft [61 m] 
decision height, 2400 ft [731.5 m] visibility) 
available throughout the United States; but only 153 
Category II (no lower than 100 ft [30.5 m] decision 
height, 1200 ft [366 m] visibility); and 118 Category 
III (a decision height lower than 100 ft, or no 
decision height, or a runway visual range (RVR) less 
than 1200 ft) IAPs. 3 Enhanced Fight Vision Systems 
(EFVS) may eliminate much of the ground 
infrastructure and charting requirements since it is 
used in the “visual” segment of the instrument 
approach. It provides a head-up, eyes out, manual 
flown procedure that is exactly the same for each 
approach and landing, regardless of the visibility 
conditions. Pilots, in essence, practice for low 
visibility operations during all takeoffs and landings 
since the visual picture and operation procedures are 
unaffected by the visibility conditions. 

EFVS is a significant part of the FAA NextGen 
Implementation Plan for improved approaches and 
low-visibility operations. 4 NASA Langley Research 
Center is conducting research to ensure effective 
technology development and implementation of 
regulatory and design guidance to support 
introduction and use of onboard Synthetic Vision 
Systems (SVS)/EFVS advanced flight deck vision 
system technologies in NextGen operations. 

Background 

SVS is a computer-generated image of the 
external scene topography that is generated from 
aircraft attitude, high-precision navigation, and data 
of the terrain, obstacles, cultural features, and other 
required flight information. EFVS is a real-time 


electronic image of the external scene generated by 
imaging sensors, such as a Forward-Looking 
InfraRed (FLIR) or Millimeter Wave Radar 
(MMWR), and presented on a head-up display 
(HUD). Both SVS and EFVS are onboard vision- 
based technologies intended to supplement or 
enhance the natural vision of the pilot in low 
visibility conditions. Combined Vision Systems 
(CVS) use a combination of SVS and EFVS for 
presentation to the pilot. 

NASA, Rockwell Collins, and others have 
developed and shown that SVS technologies provide 
significant improvements in terrain awareness and 
reductions for the potential of Controlled-Flight-Into- 
Terrain incidents/accidents, 5,6,7 improvements in 
flight technical error to meet Required Navigation 
Performance criteria, 8,9,10 and improvements in 
situation awareness without increased workload 
compared to current generation cockpit 
technologies. 11,12,13 As such, SVS, often displayed on 
a head-down display (HDD), is emerging as standard 
equipage for Part 23 and Part 25 flight decks even 
though, to date, no operational credit is obtained from 
equipage. 14 

EFVS capability on a HUD using FLIR sensor 
technology has garnered a significant share of the 
business aircraft market and is growing in Part 121 
and 135 operations. 15 EFVS provides many of the 
same operational benefits as SVS technology, but it 
uses a real-time view of the external environment, 
independent of the aircraft navigation solution or 
database. These differences, in part, enable 
operational credit with use of an approved EFVS. In 
2004, Title 14 of the Code of Federal Regulations 
(CFR) Section (§) 91.175 was amended to enable 
operators conducting straight-in instrument approach 
procedures (in other than Category II or Category III 
operations) to descend below the published Decision 
Altitude (DA), Decision Height (DH), or Minimum 
Descent Altitude (MDA), down to 100 feet above 
touchdown zone elevation (TDZE) using an approved 
EFVS in lieu of natural vision. (To descend below 
100 feet above the TDZE, the required visual 
references for landing must be distinctly visible and 
identifiable by the pilot using natural vision.) An 
approved EFVS must meet the requirements of 
§ 9 1.1 75 (m) and the use of a HUD or an equivalent 
display is essential and required by regulation. 



Synthetic and Enhanced Vision Systems (SEVS) 
technologies, such as SVS/EFVS in combination with 
HDD/HUD, form the basis for an electronic display 
of visual flight references for the flight crew. 
Integrating these SEVS displays with conformal 
symbology provides important situation, guidance, 
and/or command information as necessary and/or 
appropriate to enable all weather approach and 
landing operations. The primary reference for 
maneuvering the airplane is based on what the pilot 
sees electronically through the SEVS , in lieu of or 
supplemental to the pilot’s natural vision, in low 
visibility conditions. 

The key concept for 14 CFR §91.175 is that an 
EFVS can be used in lieu of natural vision from the 
DA/DH/MDA to 100 ft height above the TDZE 
provided the visibility of the enhanced vision image 
meets or exceeds the published visibility required for 
the approach being flown and the required visual 
references are clearly identified. Minimum aviation 
system performance standards for EFVS are available 
in RTCA DO-315. 16 RTCA DO-315 also provides 
performance standards for SVS but without 
operational credit. 

The FAA has started a rulemaking project 17 to 
expand operational credit for EFVS beyond what is 
currently authorized under 14 CFR §91.175. 
Specifically, the new regulation (14 CFR §91.176) 
will cover instrument approaches with EFVS, 
enabling landings without natural vision. The 
proposed rule would permit an EFVS to be used in 
lieu of natural vision during a straight-in precision 
instrument approach procedure (Category I, II and 
III) or an approach with approved vertical guidance. 
Current EFVS operations are limited to straight-in 
instrument approaches procedures (in other than 
Category II or Category III operations). The new rule 
would permit EFVS to be used during touchdown 
and rollout and would also add operational benefits 
for Part 9 IK, 135, and 121 operators with new 
capabilities with EFVS. 14 CFR §135.219 would be 
amended to allow dispatch to a destination that is 
below minimums if the aircraft is EFVS equipped 
and the operator is authorized for EFVS operations 
through OpSpecs (Operations Specifications), 
MSpecs (Management Specifications), or FOA 
(Letters of Authorization). Also, 14 CFR §135.225 
would be modified to enable an EFVS equipped 
aircraft and trained crew to initiate or continue an 


approach when the destination airport is below 
standard authorized minimums. Similar rules will be 
changed in CFR Part 121 to allow operation for 
scheduled air carriers. These upcoming rule changes 
support EVO through use of onboard flight-deck 
based EFVS rather than necessitating additional 
ground infrastructure equipment and operating 
procedures. An EFVS -equipped aircraft and qualified 
crew could dispatch and continue an approach to a 
landing, even when the destination airport is below 
standard authorized minimums, solely through the 
use of an electronic (sensor) image since natural 
vision is no longer required with EFVS. 18 

The FAA proposed EFVS rule change does not 
explicitly impose an RVR limitation but mentions 
that initial implementations of EFVS operations to 
touchdown and rollout may be limited to visibilities 
of no lower than 1000 RVR because airworthiness 
and certification criteria have not been developed to 
support EFVS operations below 1000 RVR. Past 
NASA simulation and flight test research 19,20,21,22 
supports the viability of this expanded EFVS 
operational credit where it was shown that using a 
single sensor EFVS (FLIR imagery on a HUD) to 
hand-fly approaches through touchdown resulted in 
excellent localizer tracking performance and an 
improvement in glideslope tracking performance in 
visibility as low as 1000 RVR. All currently 
approved EFVS use FLIR (single sensor imagery) on 
a HUD. Natural vision of 700 to 1000 ft RVR has 
been shown to be sufficient to mitigate a complete 
failure of the single sensor EFVS. FLIR is dependent 
upon atmospheric conditions. It works well in smoke, 
haze, and at night; but has degraded performance in 
fog, rain, and snow. 

RTCA DO-341 23 was drafted to establish 
minimum performance standards for EVFS 
operations on straight-in instrument approach 
procedures with published vertical guidance to 
touchdown, landing, and roll-out to a safe taxi speed 
in visibility as low at 300 ft RVR. In this operation, 
criteria are established such that the combination of 
all systems used during the EFVS operation shall be 
designed, tested, and certified to a level of safety 
appropriate for the phase of flight and the intended 
operation. As such, a fail-operational EFVS design is 
necessary - consistent with other systems and 
subsystems used for the same intended function and 
phase of flight (e.g., instrument landing systems, 



autoland systems, and navigation systems) - without 
the need or reliance of the crew’s natural vision. In 
these extremely low visibility conditions, there is not 
sufficient natural vision for the flight crew to mitigate 
certain EFVS failure conditions which might have 
catastrophic consequences. 

DO-341 specifies that both the PF and pilot- 
monitoring (PM) are required to have an independent 
EFVS HUD as well as an alternate display for 
enhanced vision (EV) sensor imagery. It is 
envisioned that more than one sensor will be needed 
for EFVS operations below 1000 ft RVR. FLIR 
sensor technology used in production EFVS is 
limited in its ability to work in extreme low visibility 
conditions, such as ground fog. Millimeter wave 
radar (MMWR) technologies show promise for 
working in operationally relevant atmospheric 
conditions, but the image resolution is not as good as 
natural vision. 

A pilot-in-the-loop high-fidelity motion 
simulation study was conducted at NASA Langley 
Research Center to evaluate the operational 
feasibility, pilot workload, and pilot acceptability of 
conducting straight-in instrument approaches with 
published vertical guidance to landing, touchdown, 
and rollout to a safe taxi speed in visibility as low as 
300 feet RVR by use of vision system technologies 
on a head-up display (HUD) without need or reliance 
on natural vision. Twelve crews flew various landing 
and departure scenarios in 1800, 1000, 700, and 300 
RVR. 

This paper details the non-normal results of the 
study including objective and subjective measures of 
performance and acceptability. Due to paper length 
restrictions, an additional paper details the normal 
results for this study. 24 

The study attempted to determine if the fail- 
operational concept detailed in RTCA DO-341 was 
operationally feasible. The concept in automated 
systems is that no single failure should cause a go- 
around below a specified altitude, called an alert 
height. For an EFVS system designed for 300 ft RVR 
conditions, the alert height was coincident with the 
decision height/decision altitude as per DO-341. 

The RTCA SC-213 committee discussed a 
number of methods that a fail operational-like system 
could be implemented with a manual flown approach. 


The committee agreed that dual HUDs, dual PFDs, 
and redundant sensors would be required. If any of 
the required EFVS systems or subsystems fail above 
the DA/DH and result in a loss of reliability, a failure 
annunciation would be given and the crew would 
execute a go-around since a fail-operational 
capability is not ensured. If any of the PF systems 
failed below the DA/DH, the fail-operational 
capability that is designed for beyond this pointwould 
mean that he/she could transfer control to the pilot 
monitoring or he/she could switch to a different 
sensor or different display and continue the landing. 
Optionally, he/she could also go-around but that 
defeats the purpose of fail-operational. 

The study was designed with dual HUD and 
dual PFD displays, all with redundant sensor 
information on the displays. Pilots were briefed and 
trained that for any failure of the HUD they could 
transition to head-down, they could transfer control 
to the other pilot, or they could go-around. They were 
to choose the safest course of action. In an attempt to 
determine how pilots would handle the failures 
without dictating specific procedures or forcing them 
to do something against their normal standard 
operating procedures, the crew were specifically not 
trained for the failures in order to get an unrehearsed 
reaction. The fail-operational discussion occurred 
twice during the pre-test briefing. 

Method 

Participants 

Twenty-four pilots were recruited representing 
various operators. Each pilot was required to hold an 
Airline Transport Pilot rating, recent HUD 
experience with at least 100 hours of HUD 
experience as pilot in command, enhanced vision 
(EVS) or EFVS experience, and glass cockpit 
experience. Crews were recruited as pairs from the 
same employer to minimize inter-crew conflicts in 
Standard Operating Procedures (SOPs) and Crew 
Resource Management (CRM) procedures or 
training. Eight crews were recruited from various 
passenger airlines, one crew was recruited from 
FedEx, and three crews were from the business 
aviation community. The business aviation crews had 
the most recent experience with EFVS. Recent HUD 
and EVS experienced ranged from “currently flying 



EFVS on the line” to “ I last used it twelve years ago 
in the military”. The Captains’ average age was 57.7 
years with an average of 1,988 HUD flight hours, 23 
years of commercial flying, and 14.8 years of 
military flying. The First Officers’ average age was 
49.7 years with an average of 1,255 HUD flight 
hours, 14.8 years of commercial flying, and 14 years 
of military flying. The Captain was the designated PF 
throughout all the trials and the First Officer served 
as the PM. 

Experimental Design 

The study used a partial 7x2x4 randomized 
block design for approaches. See Figure 1, Nominal 
Approach Matrix for a description of cross effects. 
This design resulted in 18 nominal approach 
scenarios. The crews also flew three additional 
approaches with the following failures: 

• PF complete HUD failure was presented 
using the Blended EFVS concept in 300 
RVR with 10 knots of right cross-wind on 
Runway 18F 

• PM HUD failure was presented using the 
Slant Range EFVS concept in 700 RVR 
with 10 knots of left cross-wind on 
Runway 36R 

• EFVS system failure was tested in 700 
RVR on Runway 36C with between- 
subject testing presented on either Blended 
EFVS concept or Blended with SVS EFVS 
concept 

The study used a partial 5x2 randomized block 
design for departures. See Figure 2, nominal 
departure matrix for a description of cross effects. 

The conventional HUD symbology-only baseline (no 
imagery) was tested in 300 RVR on a runway with 
centerline lighting. The EFVS concepts were tested 
without centerline lights in 300 RVR to test the 
elimination of lighting requirements for departure 
operations. An engine failure on departure was used 
to force a rejected takeoff (RTO) and this condition 
was tested twice, once with the Slant Range EFVS 
concept and once with the conventional HUD 
symbology, with the failure order counterbalanced. 
This resulted in seven departure scenarios total with 
two resulting in an RTO. 


Operational Baseline for Current 
(Single Sensor) EFVS Operations 

Testing effects of TDZ/CL lights at 
lower RVR levels of 300 and 700 ft 
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Operational Baseline for Dual Sensor EFVS Concept 
Approach Comparisons 

Testing effects of adding SVS to 
Dual Sensor Imagery 


Figure 1. Nominal Approach Matrix 
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centerline lighting requirement for HUD 
departures in 300 ft RVR 

Figure 2. Nominal 

[ Departure Matrix 


To enable the participants to more easily make 
comparisons on display concepts the RVR was 
blocked on both approach and departure runs so that 
participants were shown all concepts in one visual 
condition and then the other. The runs within each 
visibility block were randomized. The block order 
was also randomized. Total runs for each set of 
participants were 28. 

Independent Variables 

The independent variables for approach were: 

1) Seven HUD display concepts consisting of 
two baselines, two methods of displaying 
FLIR and MMWR information and three 
ways of combining synthetic vision 
information to the FLIR and MMRW 
display concepts 

2) With and without touchdown and 
centerline lighting 



3) Four visibility conditions consisting of 
1800, 1000, 700, and 300 RVR. 

The independent variables for departure were: 

1) Five HUD display concepts consisting of a 
baseline, two methods of displaying FLIR 
and MMWR information, and two 
methods of combining synthetic vision 
information to the FLIR and MMWR 
display concepts 

2) With and without touchdown and 
centerline lighting 

Figures 3 and 4 show the display conditions used in 
the study in 300 RVR with touchdown and centerline 
lighting on a representative approach at 100 feet 
above touchdown height. 



Figure 3. FLIR/MMWR EFVS HUD Concepts 
with Synthetic Vision: Blended (left side) and 
Slant Range (right side) 



Figure 4. EFVS PFD Concept 


Dependent Variables 

The dependent variables included both objective 
and subjective measures. The objective variables 
included performance in the instrument segment from 
the start of the run until decision height (DH) as 
measured by root mean square (RMS) localizer 
deviation and RMS glide slope deviation statistics; 
performance in the visual segment from DH to 
threshold crossing measured by localizer and glide 
slope deviation statistics (mean, standard deviation, 


and maximum value) at 100 feet HAT and at 
threshold crossing; percentage of missed approaches; 
touchdown performance measured by longitudinal 
distance from threshold, distance from centerline, 
sink rate statistics, and touchdown roll angle/bank 
angle to asses wing/tail strike; and rollout and 
departure performance measured by deviation from 
centerline statistics (RMS and maximum distance 
from centerline). Subjective measures included post- 
run workload using the Air Force Flight Test Center 
(AFFTC) 25 rating and Likert-type questions 26 of the 
utility of the display concepts; post-test semi 
structured debriefing questions and discussion. 
NASA Task Load Index (TLX) 27 was used for four 
approaches to gauge workload measures with more 
diagnosticity. 

Procedures 

Participants received an introductory briefing 
and informed consent, eye/head tracker calibration, 
extensive briefing of display concepts and procedures 
for EFVS approaches, simulation familiarization and 
training, two days of data collection, and a post-test 
interview and debrief. The training material included 
significant detail on all five display concepts and 
general procedures used for EFVS including required 
visual references needed to complete the approach 
using EFVS. Training consisted of cockpit 

familiarization and pointing out how the different 
image sources were affected by night and reduced 
visibility conditions. Autoland approaches were 
conducted using various visibility conditions 
encountered in the study with each of the five 
concepts. The participants were instructed to remove 
the HUD symbology and look only at the imagery for 
the autoland approaches. The crews then conducted 
hand-flown approaches with auto-throttles engaged 
for each of the five display concepts using the 
procedures and callouts expected for the study. A go- 
around was commanded by Air Traffic Control on 
the final approach run to train go-around procedures 
in the simulator. Initial training was done with 
motion off to facilitate the Principal Investigator (PI) 
standing and pointing out switches and display 
features. When the crew started hand-flying the 
approaches during training, the motion system was 
engaged. Departure training was conducted from the 
end of the runway with the last departure ending in 
an aborted takeoff initiated by the PI calling reject. 




All approaches were conducted to the Memphis 
International Airport north/south runways. 
Approaches to the north runways (36L, 36C, 36R) 
had touchdown zone, centerline, and Surface 
Movement Guidance and Control System (SMGCS) 
lighting on the ground including lead-on/lead-off 
lights and taxi centerline lighting; while approaches 
to the south runways (18L, 18C, 18R) did not. ALSF- 
2 approach lighting systems (ALS) were used on the 
north runways and MALSR ALS were used on the 
south runways. 

Data collection proceeded after the training runs 
were complete and a complete matrix of 28 runs was 
conducted. After each run, the AFFTC workload 
measures, 25 see Figure 5, and Likert-type scale 
questionnaires, 26 see Figures 6-8, were administered. 
Crews were handed sheets to quietly mark their 
individual ratings to remove experimenter and 
participant bias. Four times during the study, the 
NASA TLX 27 was administered to gather data on 
workload. The participants were queried by the PI 
and the scores were recorded. Upon completion of all 
runs, participants were debriefed with a guided 
questionnaire and semi-structured discussions. A 
pair-wise comparison was conducted 24 to evaluate the 
PF and PM’s subjective assessment of situation 
awareness for each of the five EFVS concepts flown 
in extremely low visibility conditions. The PF and 
PM were also asked to rank order the 5 EFVS 
concepts they evaluated in terms of preference for 
flying with in low visibility conditions. 



AFFTC Workload Estimate 

(On the sheet provided, write the number of the statement which best describes the 
MAXIMUM workload you experienced during the past work period.) 

1 

Nothing To Do; No System Demands 

2 

Light Activity; Minimum Demands 

3 

Moderate Activity; Easily Managed; Considerable Spare Time 

4 

Busy; Challenging But Manageable; Adequate Time Available 

5 

Very Busy; Demanding To Manage; Barely Enough Time 

6 

Extremely Busy; Very Difficult; Non-Essential Tasks Postponed 

7 

Overloaded; System Unmanageable; Important Tasks Undone 


Figure 5. Air Force Flight Test Center Workload 
Scale 


Post-Run APPROACH 
Ratings 

On the sheet provided, write the number 
which bests corresponds to your rating of 
agreement with each statement based on 
the HUD EFVS concept you just 
evaluated 

Strongly 

Disagree 

1 

Disagree 

2 

Slightly 

Disagree 

3 

Neither 
Agree or 
Disagree 

4 

Slightly 

Agree 

5 

Agree 

6 

Strongly 

Agree 

7 

Q1 . Prior to DH, 1 was able to 
pick up the necessary visual 
references to continue the 
approach or landing. 








Q2. The landing visual 
references were visible and 
identifiable no lower than 100 
ft AFL (i.e., the runway and 
touchdown zone). 








Q3. There was sufficient time 
to recognize and identify the 
required visual references. 








Q4. The visual information 
provided sufficient cues to 
flare and land. 








Q5. 1 was able to maintain 
lateral alignment with the 
runway. 








Q6. 1 was able to complete the 
approach and land safely. 









Figure 6. Post-Run Approach Questionaire 


Post-Run TAXI 
Ratings 

Strongly 

Disagree 

Disagree 

Slightly 

Disagree 

Neither 
Agree or 
Disagree 

Slightly 

Agree 

Agree 

Strongly 

Agree 

On the sheet provided, write me 
number which bests corresponds 
to your rating of agreement on me 
HUD EFVS concept you just 
evaluated. 

1 

2 

3 

4 

5 

6 

7 

Q1 . EFVS improved my 
ability to taxi. 








Q2. The Field-of-Regard 
of the EFVS was 
sufficient to taxi. 








Q3. 1 was able to safely 
taxi the aircraft. 









Figure 7. Post-Run Taxi Questionaire 


Post-Run 

DEPARTURE 

Ratings 

Strongly 

Disagree 

Disagree 

Slightly 

Disagree 

Neither 
Agree or 
Disagree 

Slightly 

Agree 

Agree 

Strongly 

Agree 

On the sheet provided, write Ihe 
number which bests corresponds 
to your rating ot agreement on 
the HUD EFVS concept you just 
evaluated 

1 

2 

3 

4 

5 

6 

7 

Q1. 1 was able to 
position the aircraft on 
runway centerline using 
the EFVS in preparation 
for takeoff roll. 








Q2. 1 was able to 
maintain centerline 
during takeoff roll. 








Q3. 1 was able to 
maintain runway 
heading during initial 
climbout. 








Q4. 1 was able to safely 
complete the departure. 









Figure 8. Post-Run Departure Questionaire 


Apparatus 

The study was conducted in the NASA Langley 
Research Center (LaRC) Research Flight Deck 
(RFD) simulator. The study was conducted in full 
motion with the RFD articulated on top of a hexapod 
hydraulic motion system. The RFD is representative 







of a state-of-the-art advanced subsonic transport 
airplane with fully reconfigurable flight deck 
systems. The RFD is composed of a 200 degree x 40 
degree field-of-view out-the-window collimated 
scenery system. The flight deck includes three large 
landscape format Liquid Crystal Displays configured 
as two primary flight displays, two navigation 
displays and engine instruments and crew alerting 
system (EICAS) displays, cursor controllers, and 
multifunction controls on the center control stand. 
Electronic flight bag (EFB) displays are outboard on 
each pilot’s side. Movable driven throttle system, 
including auto-throttles, is implemented. Active side 
stick controllers coupled side to side are provided for 
manual aircraft control. Dual HGS-6700 head-up 
displays are fitted with overhead brightness controls. 
A castle switch on the side-stick controls image and 
symbology on the HUD. Concepts to be studied were 
displayed on the both HUDs and PFDs as shown in 
Figure 9. Checklists and approach charts were 
available on the EFB. 



Figure 9. Research Flight Deck 

Three detailed databases were developed for the 
study. 24 First a detailed out-the-window visual 
database was created for the Memphis airport and 
surrounding area. All of the terrain features down to 
the foot level were modeled as well as roads and 
airport structures. From this detailed database, two 
additional databases were created using material 
codings for all objects to associate infrared (IR) and 
radar properties onto each feature. The IR database 
was rendered using standard tools developed by 
Rockwell Collins Simulation and Training Systems, 
modeling a short-wave/mid-wave cooled IR sensor. 
The MMWR simulation was rendered using a 


physics-based modeling process developed at NASA 
LaRC. This model produced a detailed 3D MMWR 
image that was mapped to a B scope image and then 
mapped back to lower resolution C scope image. 
Radar parameters simulated a 94 GHz radar with a 5 
mile range, using 5 meter range bins and 0.5 deg 
angular width, simulating emerging 3D MMWR 
systems. 

Results 

Statistical results were evaluated in Minitab. 

All landings, both nominal and those conducted 
during failure conditions, were within visual 
performance standards of the first third of the runway 
and within the lateral confines of the runway. All but 
three of the approximately 250 landings were within 
Category III autoland tolerance as defined in AC 
120-28D 26 and the All Weather Operations manual. 27 
Of those three, two were firm landings from the same 
crew in failure conditions and one was a landing too 
near the threshold in the baseline HUD symbolgy- 
only case in visual conditions of 1800 RVR. 

No go-arounds occurred for any of the 
approaches except in response to failures and one 
scenario where a crew decided that traffic was still on 
the runway while on short final. 

Pilot-Flying HUD Failures 

To test the fail-operational concept of the 
operation and to determine crew response to failures, 
the pilot-flying HUD was randomly failed at 80 feet 
height above the threshold. A Blended EFVS concept 
in 300 RVR to Runway 18L with 10 knots of left 
cross-wind was used. Proper crew response would be 
to go-around, continue head-down, or transfer control 
to the pilot monitoring to complete the landing. All 
but one of the crews conducted a go-around and 
Figure 10 shows the go-around results. Nominal 
threshold crossing was 50 feet and the failure 
occurred at 80 feet. Mean altitude when the TOGA 
button was pressed was around 80 feet and minimum 
altitude attained averaged around 45 feet. These 
altitude values represent a quick determination by the 
PF and a positive climb rate not long after threshold 
crossing. Eleven crews executed a timely go-around 
with no excursions from centerline tracking. 




Figure 10. Box Plot of TOGA Altitude, Threshold 


Altitude, and Minimum Altitude 

Table 1 shows the mean subjective ratings for 
PF and PM in response to the post-run questions 
(Figure 6). PFs rated their workload as being 
challenging but manageable during the complete 
failure of their HUD at 80 ft AGL. PMs rated their 
workload as being easily handled during this failure 
of the PF’s HUD. Both crew members slightly agreed 
that prior to DH they had sufficient time to recognize 
and identify the required visual references to continue 
the approach or landing in 300 RVR. The PFs agreed 
that the runway and touchdown zone required visual 
references were visible and identifiable by 100 ft 
above field elevation. 



PF ratings 

PM ratings 

AFFTC 

Workload 

4.0 

3.1 

Approach Q1 

5.3 

5.1 

Approach Q2 

5.7 

4.8 

Approach Q3 

5.3 

4.8 


Table 1. Workload and Post-Run Approach 
Questionnaire Ratings for PF HUD Failure 


landing rollout. The PM during the landing rated his 
workload as challenging but manageable, slightly 
agreed that he was able to complete the approach and 
landing safely, and agreed that the required visual 
references were available and adequate and he could 
maintain lateral alignment with the runway. 

Pilot Monitoring HUD Failures 

Since the pilot monitoring HUD failure was 
equivalent to a failure of the standby channel it was 
basically a non-event. No go-arounds were conducted 
for this failure and all 12 crews performed a 
successful landing. The failure was presented on the 
Slant Range concept in 700 RVR with 10 knots of 
right cross-wind to Runway 36R. To test for 
equivalent performance, a comparison was made with 
the exact same conditions except no failure and 5 
knots of left cross-wind. Figure 1 1 shows lateral and 
vertical RMS errors in feet from decision height to 
touchdown. There was no significant (p>0.05) 
differences in the vertical or lateral errors between 
the EFVS failure condition and the same EFVS 
concept with no failure. 


Interval Plot of Lateral RMS Feet, Vertical RMS Feet 

95% a for the Mean 



Individual standard deviations are used to calculate the intervals 


Figure 11. PM HUD Failure RMS Errors 


One of the crews discussed the transfer of 
control idea over the course of the study and 
completed the transfer of control to a landing. With a 
sample size of one, only generalizations can be made 
with the data. Even just comparing the data within 
the crew doesn’t work because the PM only made 
this one landing. An additional complicating factor is 
on this particular approach the pilot flying was 15 
feet left of centerline at decision height. The landing 
was made well within the autoland criteria. There 
was up to 15 feet maximum lateral error during the 


Table 2 details subjective ratings for PF and PM 
during the failures. Both crew members rated their 
workload as being easily handled with adequate time 
available during this event. The PFs agreed that they 
had sufficient time to recognize and identify the 
required visual references to continue the approach or 
landing and the imagery provided them sufficient 
information to flare, land and maintain centerline in 
700 RVR. The PFs strongly agreed that they were 
able to complete the approach and land safely even in 
the presence of a complete PM HUD failure. 





PF ratings 

PM ratings 

Workload 

3.0 

2.7 

Approach Q1 

6.2 

5.8 

Approach Q2 

6.2 

5.3 

Approach Q3 

6.2 

5.7 

Approach Q4 

6.3 

5.2 

Approach Q5 

6.4 

5.9 

Approach Q6 

6.6 

5.9 


Table 2. Workload and Post-Run Approach 
Questionnaire Ratings for PM HUD Failure 


EFVS System Failures 

Pilots were briefed that EFVS failures would be 
annunciated on the HUD and would trigger an 
EICAS caution alert. The response to a failure above 
decision height was a go-around. Failures below 
decision height did not require a go-around. For this 
study, the “EFVS Fail” message was triggered at 350 
feet, which is above decision height, and the proper 
response would have been a go-around sometime 
before the decision height. The failure was an 
insidious failure since the system continued to 
operate normally until 80 feet where a complete 
EFVS system failure (symbology and imagery 
removed from HUD) occurred for both the PF and 
PM. “EFVS FAIL” was presented using the blended 
FLIR/MMWR concept, with and without synthetic 
vision, in 700 RVR with 15 knots of left cross-wind 
to Runway 36C. Seven of twelve crews responded 
correctly and conducted a go-around before decision 
height. The performance is detailed in Figure 12. One 
crew didn’t conduct a go-around until the second 
failure at 80 feet and it was a successful maneuver. 
Four of the twelve crews disregarded the second 
failure and completed the landing with a blank HUD 
using available (700 RVR with touchdown zone and 
centerline lighting) out the window visibility cues. 
Since there are a limited number of landings and this 
landing case represent an operational scenario that 
cannot be allowed to happen, no statistical analysis 
was conducted for the landings. 



Figure 12. Interval Plot of TOGA Altitude, 
Minimum Altitude, and Threshold Altitude for 
EFVS Failure 

Crews rated their workload (mean=3.5) as 
requiring moderate activity to being busy during the 
annunciated EFVS HUD failure. 

Departures 

Take-offs were conducted in 300 feet of 
visibility with five knots of crosswind. A symbology 
only baseline was tested with runway centerline 
lighting as currently allowed under special 
authorization. EFVS concepts were tested with no 
runway centerline lighting. Failure conditions 
consisted of left engine failures at 100 knots using 
symbology only and Slant Range concepts. Figure 13 
shows the absolute value of maximum distance from 
centerline and RMS values of centerline deviations 
(both measured in feet) during takeoff to gauge 
centerline tracking. A paired T test between failure 
conditions showed no significant difference 
(p=0.135) but a paired T test between each failure 
condition and the EFVS concepts with no failures 
showed a significant difference (p=0.006 for each 
pair) and a significant difference between the 
symbology only baseline (p=0.004). Centerline 
tracking for the normal departures showed no 
significant (p>0.05) differences compared to the 
symbology only departure even though the EFVS 
departures did not have centerline lighting. None of 
the departures had excursions beyond operational 
significance and any lateral excursions during engine 
failures were quickly corrected to centerline. 



Interval Plot of Centerline Tracking Performance 

95% a for the Mean 

SR - Slant Range. C - Combined, Sym - Symbology S/ - with SVS F - Fail 



Figure 13. Departure Performance in Feet from 
Centerline 


Having imagery during a rejected takeoff 
appears to have slightly decreased the PF’s workload. 
With imagery, PFs rated (mean workload=3.4) the 
task as requiring moderate activity but with 
considerable spare time to attend to other tasks. 
Without imagery, the PFs rated their workload 
(mean=3.8) as being challenging but manageable and 
with adequate time available to attend to other tasks. 
Comparable workload for the PM was found with 
either the Conventional HUD or Slant Range HUD as 
PMs rated their workload (mean=3.0) as being easily 
managed during the rejected takeoff runs. 


Discussion 

Universally, all pilots had high praise for the 
EFVS concepts. All concepts worked equally well. 
Although this paper details the non-normal 
conditions, a short discussion of the normal landings 
is provided with no supporting data. There was a 
pilot preference for a combination of synthetic vision 
combined with sensor information but there was no 
difference in performance with or without synthetic 
vision for the landing tasks. There was a tight 
grouping of touchdown dispersions. No difference 
was noted between the 700 foot or 300 foot visibility 
cases. The data supports RTCA DO-341 and the 
operational feasibility of approach, landing, and 
rollout in visibilities as low as 300 RVR. 
Performance was at least as good as or better than 
currently approved EFVS operations in higher 
visibilities (1000 RVR with FLIR) and better than the 
symbology only (1800 RVR) case with touchdown 
zone and centerline lighting infrastructure required. 


The study attempted to determine if the fail- 
operational concept detailed in RTCA DO-341 was 
operationally feasible. Pilots were briefed and trained 
that for any failure of the HUD they could transition 
to head-down, they could transfer control to the other 
pilot, or they could go-around. They were to choose 
the safest course of action. 

In the event of a PF HUD failure, 11 of the 12 
crews performed a go-around. While safe, this 
procedure somewhat defeats the purpose of a fail- 
operational system design. Only one crew picked up 
on the discussion enough to continue the discussion 
amongst themselves. They were a business jet crew 
where the authority gradient is often more level than 
the airlines. They received the PF HUD failure on the 
first day and at the start of the second day had clearly 
discussed it over the evening. They asked if the pilot 
monitoring could perform one training landing, and 
the request was granted. When they received the 
failure, they transferred control and performed a 
successful landing. Although there was a slight 
excursion from centerline during the transition, the 
landing was well within autoland certification 
criteria. Since only one landing was attempted, no 
real conclusion can be drawn from the objective data. 
The PF was 15 feet left of centerline at decision 
height, the PM landed within 2 feet of centerline and 
then crossed 15 feet right of centerline on rollout 
before correcting back to centerline. The entire 
excursion could be due to the initial lateral error at 
DH. The other eleven crews conducted a smooth and 
immediate go-around in response to the failure. No 
crews attempted to transition head-down and since 
they weren’t trained for that transition, this result was 
not unexpected. 

During post-run interviews, all pilots 
commented that transferring control to the other pilot 
at low altitude was not allowed in standard operating 
procedures. Many pilots indicated that airline 
procedures would not allow transfer of control below 
500 feet and some even indicated not below 1,000 
feet. Airline pilots commented that typically low 
visibility approaches would be flown by the captain 
and they would not transfer control to the co-pilot. 
Since neither a monitored approach (Captain 
monitors a co-pilot flying) nor transfer of control to a 
co-pilot is acceptable, the ability to transition head- 
down to the PFD with a sensor view is something 
that will have to be explored in the future. The PM 


HUD failure was essentially a failure in the standby 
channel of the total system. Except for the distraction 
of the monitoring pilot announcing a failure, 
performance was as good as the same condition 
without a failure. The only difference was in the wind 
condition where no failure was 10 knots of left 
cross wind and the failure condition was in 10 knots 
of right crosswind. Figure 11 details RMS errors in 
the visual segment from 200 feet above the runway to 
touchdown. Using RMS errors to remove the wind 
effects, there was no performance difference for the 
failure compared to the same EFVS concept with no 
failure in either the vertical or lateral dimensions. 
Subjective ratings support no difference in 
performance between failures and non-failures in the 
monitoring pilot. This does support the fail-operation 
conclusion for monitoring failures. 

EFVS system failures were not always handled 
correctly. Although the failures were discussed in 
training and briefings, the failures were not trained or 
practiced in the simulator. Seven of twelve crews 
performed a correct and successful go-around, see 
Figure 12. One of the crews performed a go-around 
around 70 feet above the runway when the image 
blanked for the second failure and four crews decided 
at the second failure that they had sufficient visual 
cues to complete a successful landing in 700 foot 
visibility with touchdown zone and centerline 
lighting. The EFVS fail annunciation on the HUD did 
not flash or become boxed as normal because the 
software could not be easily changed. The EICAS fail 
annunciation was near the top of the EICAS display 
and a number of crews said it was too high to be 
easily seen. All of these items could have been 
covered in training but the experimenters felt this 
would eliminate the element of surprise when the 
failure occurred. For an operational system, the 
design methodology and crew procedures for a fail- 
operational vision system would have to be well- 
engrained in training. 

Normal take-off tracking was good for all 
concepts, see Figure 13. Centerline tracking 
performance did not degrade for EFVS with no 
centerline lighting compared to symbology-only with 
centerline lighting. 

There was a significant effect in centerline 
tracking performance during engine failures at 100 
knots. This result was not unexpected since just the 


initial yawing from the engine failure alone would 
cause some lateral errors. Successful certification 
criteria during failures for departure is that the 
aircraft must not depart the runway and tracking back 
to centerline is smooth after any excursions. All of 
the failure departures resulted in all of the crews 
correctly determining that the engine had failed with 
an appropriate rejected takeoff below VI. None of 
the failure cases resulted in an excursion off the 
runway. There was no significant difference between 
the EFVS failures on a runway with no centerline 
lighting and the baseline symbology-only case with 
centerline lighting. Subjectively, pilots indicated 
departure operations using EFVS was operationally 
feasible. This performance was probably, in fact, 
conservative because of an unexpected simulator 
mechanization fault in the hexapod motion base and 
the brake models. The rejected takeoff and normal 
maximum manual braking moved the hexapod to the 
forward stop during the initial deceleration. This 
action caused a sudden stop and shudder in the flight 
deck and produced a significant startle effect that 
happens right during the time that rudder corrections 
needed to be made to return to centerline. The result 
was an initial removal from the brakes and rudder 
pedals that probably resulted in more lateral 
excursion than normal. 

Conclusion 

This study supports the operational feasibility of 
conducting approach, landing and rollout to a safe 
taxi speed in visibilities as low as 300 RVR using 
Enhanced Flight Vision Systems (EFVS). All twelve 
crews universally decided that the system provided 
all the visual cues required in the visual segment at or 
before the decision height to continue for a landing. 
Many crews had the required references as early as 
500 feet. Most crews commented they wanted this 
system immediately and it was a significant 
improvement over current low visibility landing 
systems. Success was independent of visibility. All 
concepts of EFVS, with and without synthetic vision, 
performed equally well. Landing performance was 
independent of approach lighting and touchdown 
zone and centerline lighting systems. Performance 
was improved over current Special Category I 
operations of HUD symbology with 1800 RVR with 
touchdown zone and centerline lighting, even in 300 
RVR conditions with no centerline lighting. 



The fail-operational concept needs further study. 
It is clear that depending on the pilot monitoring to 
take control during failures is not operationally 
feasible - at least, as it is currently trained and 
implemented in today’s low visibility approach 
procedures. Using the head-down display, if designed 
correctly, may provide a fail-operational path for the 
PF. Additionally, the HUD for the PM provides the 
capability to call excursions during failures and to 
safely complete the operation during pilot 
incapacitation or landing errors. This was not studied 
and would require additional investigation. Fail- 
passive operations, potentially even in 300 RVR, 
could be feasible since the pilot is actively in the loop 
hand flying and a go-around is a natural response to 
any detected failure. The failures chosen for the study 
were felt to be the worst failures to handle. They 
resulted in a complete loss of function of visual and 
symbology elements by failing the complete HUD. 
This complete loss of information did result in an 
immediate and easily recognizable failure, so 
additional study is required using more subtle 
failures. The authors have evaluated frozen images 
and even those failures are immediately recognizable 
by pilots but all potential failures would need to be 
evaluated. Training on system failures and the proper 
procedural aspects of those failures is required. A 
number of crews ignored failure messages. 
Operationally all failure annunciations on the HUD 
should flash and be boxed to capture the pilots 
attention. This study did not do that because of 
software change limitations. 

Departures were made safely using EFVS 
concepts, even on runways with no additional 
lighting infrastructure, with an equivalent level of 
safety as compared to today’s operations in both 
normal and failure conditions. This outcome may 
allow future operations using EFVS systems for 
departure credit. Engine failures were evaluated at 
100 knots but no failures were evaluated at or beyond 
VI. It is not expected that failures at or above VI 
would be a problem but is something that requires 
future study to ensure there are no issues. No failures 
were evaluated in the EFVS or HUD images for 
departures and would require further study. 

Taxi was not an investigative part of the study, 
but taxi operations were observed for runway exit 
and turnoff toward correct taxi clearance as well as 
taxi to the runway during departure scenarios. Crews 


described taxi operations as normal and they were 
able to perform turns even with significant oversteer 
in 300 RVR. 
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